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Abstract:  In  February  2010,  Canada  hosted  the  Vancouver  2010  Winter  Games.  To  ensure  a  Safe 
and  Secure  Games,  multiple  safety  and  security  agencies  at  three  levels  of  government  had  to  work 
in  an  unprecedented  partnership  for  security  planning  and  operations.  This  whole-of-government 
approach  to  domestic  safety  and  security,  often  bringing  together  non-traditional  partners,  provided  a 
unique  opportunity  to  analyze  collective  strengths  and  weaknesses  and  to  make  suggestions  for 
process  improvements  in  future  domestic  inter-organizational  public  safety  or  security  events. 
Defence  Research  and  Development  Canada  -  Centre  for  Security  Science  undertook  an  After  Event 
Review  which  incorporated  qualitative  operational  research  methods  into  a  Lessons  Learned  process. 
Typically  a  Lessons  Learned  cycle  involves  five  steps:  preparation,  collection,  analysis,  endorsement, 
and  change.  Often,  the  process  relies  upon  initial  observations  without  the  benefit  of  independent 
analysis.  The  After  Event  Review  refined  the  preparation,  collection  and  analysis  stages  by  focusing 
on  analytical  methodologies  and  inserting  subject  matter  expert  validation  throughout  the  process. 
Beginning  with  a  mission  analysis  in  the  preparation  phase,  the  researchers  were  able  to  determine 
the  expected  outcomes  against  which  to  appraise  the  ability  of  the  partners’  collective  success.  It  was 
also  necessary  to  determine  a  capability  framework  against  which  to  assess  the  overall  planning  and 
operations.  For  this  purpose,  the  US  Department  of  Homeland  Security’s  Target  Capability  List  was 
selected.  During  the  collection  phase,  interview  surveys,  a  social  network  analysis  and  case  studies 
were  employed  and  resulted  in  the  identification  of  nine  critical  issues  for  analysis.  Using  the 
capability  assessment  approach,  the  researchers  were  then  able  to  identify  best  practices  and 
corrective  actions  which  could  be  applied  to  future  domestic  security  operations.  Subject  matter 
experts  were  consulted  throughout  to  determine  if  the  findings  were  indicative  of  operational  realities. 
The  resulting  conclusions  could  lead  to  recommendations  for  the  consideration  of  change  authorities 
within  the  federal  security  community.  The  enhanced  Lessons  Learned  approach  gave  the  final 
results  a  level  of  validation  that  may  not  normally  be  recognized  through  a  less  thorough  process. 

Keywords:  Lessons  Learned,  After  Event  Review,  Social  Network  Analysis,  V2010,  Security 
Planning,  Operational  Research 

1.  Background 

In  February  2010,  Canada  hosted  the  Vancouver  2010  Winter  Games  (V2010).  To  ensure  a  “safe 
and  secure  Games,”  multiple  safety  and  security  agencies  at  three  levels  of  government  had  to  work 
in  an  unprecedented  partnership  for  security  planning  and  operations.  This  whole-of-government1 
approach  to  domestic  safety  and  security,  often  bringing  together  non-traditional  partners,  provided  a 
unique  opportunity  to  analyze  collective  strengths  and  weaknesses.  Defence  Research  and 
Development  Canada  -  Centre  for  Security  Science  (DRDC  CSS)  undertook  an  After  Event  Review 
(AER)  which  incorporated  qualitative  operational  research  methods  into  a  Lessons  Learned  process. 
The  objective  of  the  AER  was  to  identify  and  analyze  critical  issues  common  to  the  federal  security 


1  “Whole  of  government”  denotes  public  service  agencies  working  across  portfolio  boundaries  to  achieve  a 
shared  goal  and  an  integrated  government  response  to  particular  issues.  Approaches  can  be  formal  and  informal. 
They  can  focus  on  policy  development,  program  management  and  service  delivery.”  Australian  Public  Service 
Commission,  2004. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
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partners  and  to  make  suggestions  for  best  practices  and  process  improvements  in  future  domestic 
inter-organizational  public  safety  or  security  events. 

The  objective  of  a  Lessons  Learned  process  is  to  achieve  performance  improvement.  Lessons  are 
only  “learned”  through  an  intentional  process  which  results  in  changed  doctrine,  operating  procedures 
and  behaviour.  Organizational  learning  is  challenging  for  most  institutions  and  is  only  made  more 
complex  in  meta-organizational  (i.e.,  an  organization  of  organizations)  situations  as  is  experienced  in 
whole-of-government  collaboration.  One  of  the  greatest  challenges  in  making  a  Lessons  Learned 
process  an  effective  organizational  learning  tool  is  in  how  to  perform  an  analysis  that  produces 
reliable  results.  The  AER  incorporated  operational  research  and  analysis  techniques  into  a  Lessons 
Learned  process. 

2.  Lessons  Learned  Cycle 

The  Lessons  Learned  Cycle  is  based  upon  the  premises  of  continuous  learning  as  demonstrated  in 
the  “OODA  Loop”  (Boyd,  1987).  The  four  steps  of  the  loop  are  to: 

•  Observe  actions  and  their  effects; 

•  Orientate  and  analyze  them  to  determine  the  basic  causes  and  impacts  and  consider  options 
for  behavioural  or  systemic  change; 

•  Decide  by  accepting  or  rejecting  the  options;  and 

•  Act  by  implementing  the  preferred  course  of  action. 

The  OODA  Loop  is  essentially  a  problem  solving  and  decision  making  model  that  allows  iterative  but 
deliberate  change.  Each  of  these  steps  can  be  embedded  into  a  broader  Lessons  Learned  process 
as  illustrated  in  Figure  1  (adapted  from  Eaton  et  al.,  2006). 


r 

1 

:  ' 

Jreparatior 

■  , 

J. 

■ 

r 

~  t 

L  11 

indorse  ant 
iror.t  Chang 

C  j 

1 

Collection 

^A 

Analyze 

k 

A 

Figure  1:  Lessons  Learned  Cycle 

The  Lessons  Learned  Cycle  normally  begins  in  anticipation  of  an  event,  such  as  an  exercise  or 
operation,  as  a  result  of  a  milestone  in  a  process,  or  in  the  aftermath  of  an  event.  The  process  has  six 
steps:  preparation,  collection,  analysis,  endorsing  and  directing  change,  implementation,  and 
monitoring  and  validating  change. 

2.1  Preparation 

Preparation  includes  identifying  the  functions  and  activities  to  be  analyzed,  the  critical  issues  to  be 
explored,  and  the  method  and  details  of  data  collection.  In  anticipation  of  the  event,  a  change  leader 
(the  person  responsible  for  the  guiding  documentation,  process  or  event)  will  identify,  in  consultation 
with  other  planners,  the  main  issues  which  should  be  targeted  for  observation,  e.g.,  are  the 
communications  protocols  between  organizations  functioning  as  required?  Once  the  main  issues  are 
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identified,  the  methods  of  data  collection  should  be  identified  and  prepared.  This  stage  should  involve 
a  mission  analysis  to  determine  the  objectives  of  the  process  or  event. 

2.2  Collection 

Collection  is  the  process  of  observing,  collecting  data,  and  identifying  lessons.  This  involves  the 
actual  data  collection  during  or  post-event  by  employing  either  observers  or  participants.  Collection 
can  occur  through  observation  and  check-lists,  surveys,  interviews,  data  collection  and  after  action 
reviews.  Ideally,  it  can  occur  during  the  event,  but  may  require  post-event  collection  as  well. 

2.3  Analysis 

Analysis  is  the  systematic  identification  of  issues,  the  objective  analysis  of  the  data,  and  the  validation 
of  results  with  subject  matter  experts  (SMEs).  Whether  scientifically  rigorous  or  not,  analysis  is 
important  in  processing  subjective  observations  and  dismissing  the  spurious  assumptions  to  find 
authentic  solutions.  An  important  portion  of  this  process  is  validation,  whereby  SMEs  at  various  levels 
determine  if  changes  and  solutions  are  workable.  This  can  be  accomplished  through  individual 
consultation  or  in  an  organized  conference  of  stakeholders. 

2.4  Endorsing  and  Directing  Change 

Endorsing  and  directing  change  is  the  stage  at  which  recommendations  for  corrective  actions  and 
best  practices  are  made  to  the  change  authorities  (decision  makers),  who  in  turn  accept  or  reject  the 
proposed  changes.  This  occurs  when  the  change  authority  considers  the  options  and  either  rejects  or 
endorses  recommendations.  It  is  then  the  responsibility  of  the  change  agent  or  other  officials  to  enact 
the  change. 

2.5  Implementation 

Implementation  involves  integrating  the  identified  lessons  into  existing  documentation,  training,  and 
processes.  This  systemic  change  should  beget  behavioural  change.  During  implementation,  it  is  the 
responsibility  of  the  designated  or  multiple  change  agents  to  initiate  and  manage  the  recommended 
best  practices  or  corrective  measures. 

2.6  Monitoring  and  Validating  Change 

Monitoring  and  validating  change  is  the  confirmation  that  the  recommendations  have  been 
implemented  and  verification  that  the  changes  are  effective.  The  change  agent  reports  back  to  the 
decision-maker  and  stakeholders  as  to  the  implementation  and  effectiveness  of  the  change.  It  is  at 
this  stage  that  the  Lessons  Learned  Cycle  begins  again. 

3.  Methodology 
3.1  Mission  Analysis 

Clarifying  mission  objectives  is  an  important  aspect  of  the  preparation  stage  in  a  Lessons  Learned 
process.  It  provides  a  baseline  from  which  to  ascertain  whether  the  mission  was  successfully 
achieved  and  what  facilitated  or  hindered  that.  It  also  enables  stakeholders  to  learn  how  to  better 
perform  in  future  situations.  In  the  case  of  whole-of-government  security  planning  for  V2010,  the 
mission  was  clear:  a  Safe  and  Secure  Games.  The  mission  essential  components,  (indicators  of  a 
successful  mission),  or  the  operational  objectives  from  a  whole-of-government  point  of  view, (how  the 
mission  was  to  be  accomplished),  were  not  as  clearly  articulated. 

It  was  necessary,  therefore,  to  retrospectively  determine  what  these  components  and  objectives  were 
in  order  to  frame  the  lessons  learned  process  overall.  For  example,  one  mission  essential  component 
was  deemed  to  be  the  need  to  provide  a  safe  and  secure  environment  for  a  successful  international 
sporting  event  -  including  regional  celebrations  and  the  protection  of  Internationally  Protected 
Persons  and  other  dignitaries.  Likewise,  an  operational  objective  for  the  whole-of-government 
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approach  would  be  to  test  and  confirm  federal  operational  interoperability  and  readiness  for  V2010. 
This  mission  analysis  created  the  structural  basis  for  scoping  the  overall  project. 

3.2  Critical  Issues 

The  first  step  in  the  methodology  of  the  AER  process  was  the  identification  of  critical  issues,  which 
further  aided  the  development  and  structuring  of  the  rest  of  the  methodology,  as  well  as  guided  and 
focused  data  collection.  In  order  to  identify  the  critical  issues,  V2010  planning  phase  documentation, 
lessons  learned  reports  from  the  exercise  program,  and  post-event  reports  from  individual  partner 
departments  and  agencies  were  reviewed  and  analyzed.  Through  this  process,  five  main  critical 
issues  arising  out  of  federal  whole-of  government  security  planning  and  operations  were  identified  for 
exploration.  These  included  such  issues  such  as  information  sharing  between  partners.  Based  on 
the  critical  issues,  research  questions  were  developed.  The  questions  were  used  to  guide  the 
structured  portion  of  the  interviews  which  followed. 

3.3  Data  Collection 

The  data  collection  was  comprised  of  four  analytical  processes: 

•  Collection  of  documentation  and  reports  from  the  planning  phase,  the  exercise  program,  and 
post-event.  These  were  primarily  used  to  guide  the  identification  of  critical  issues  and  later  to 
validate  preliminary  findings. 

•  Interviews  of  senior  officials  who  occupied  pivotal  positions  at  the  strategic-operational 
interface  during  V2010  planning  and  operations. 

•  Collection  of  social  network  analysis  data.  The  first  phase  data  was  collected  as  part  of  the 
interviews  while  second  phase  data  was  collected  by  means  of  a  web-based  survey. 

•  Collection  of  case  studies  data  in  three  domain  areas  of:  Critical  Infrastructure  Protection, 
Chemical,  Biological,  Radiological,  Nuclear  and  Explosives  (CBRNE)  preparedness,  and 
inter-agency  cooperative  information  sharing  as  examined  in  the  Olympic  Marine  Operations 
Centre.  The  case  study  data  was  collected  as  part  of  the  initial  interviews  as  well  as  through 
follow-up  interviews  of  key  stakeholders  who  were  directly  involved,  at  both  the  strategic  and 
operational  levels,  in  each  of  the  three  domains. 

3.4  Assessment  Criteria 

To  provide  scientific  rigour  to  the  data  analysis,  appropriate  benchmarks  were  required  against  which 
results  could  be  assessed.  Currently,  Canada  does  not  have  comprehensive,  accepted  and 
standardized  national  benchmarks  against  which  to  assess  capabilities.  Given  this,  the  Target 
Capabilities  List  (TCL)  developed  by  the  Federal  Emergency  Management  Agency  was  used  as  a 
benchmark  for  the  assessment  (US  Department  of  Homeland  Security,  2007).  In  instances  where  the 
subject  was  not  within  the  scope  of  the  TCL,  other  relevant  best  practices,  extracted  from  literature, 
were  applied  in  the  assessment  (Lemyre,  2010) 

3.5  Limitations  and  Constraints 

It  is  important  to  acknowledge  the  limitations  and  constraints  of  the  methodology  and  process  which 
were  implemented  in  the  AER.  The  main  constraint,  given  a  study  with  such  a  broad  scope,  was 
keeping  with  tight  time-deadlines  while  at  the  same  time  ensuring  scientific  rigour  and  integrity.  The 
time  deadlines  were  implemented  to  keep  the  information  current  and  relevant  and  to  ensure  timely 
feedback  to  the  relevant  stakeholders.  However,  given  the  tight  timelines,  not  all  issues  were  explored 
in-depth  and  instead  the  AER  focused  on  providing  a  broad  overview  of  the  main  issues. 
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The  second  limitation  was  associated  with  the  sole  reliance  on  post-event  data.  Since  the  study  was 
initiated  after  V2010,  this  precluded  any  opportunity  for  in  situ,  real-time  data  collection  during 
operations.  This  limitation  meant  that  the  analysts  had  to  rely  on  the  interviewees’  recall,  and 
therefore  a  possibility  existed  that  some  issues  may  have  been  missed  simply  because  the 
interviewees  may  have  forgotten  about  them.  This  was  mitigated  by  the  relatively  large  number  of 
interviewees,  many  with  overlapping  experiences,  and  by  employing  data  validation  methods  as 
outlined  below. 

Finally,  since  the  analysis  employed  in  this  study  consisted  largely  of  the  synthesis  of  the  experiences 
and  opinions  expressed  by  the  interviewees,  there  was  a  danger  of  “confirmation  bias”,  or  the 
tendency  to  form  early  opinions  and  selectively  favour  data  that  validated  those  opinions.  In  order  to 
avoid  this  phenomenon,  the  analysts  continuously  challenged  one  another’s  assumptions  and 
collectively  identified  any  major  themes  arising  from  the  interviews,  thus  eliminating  issues  which  were 
not  broadly  acknowledged.  Furthermore,  where  possible,  analysis  and  conclusions  were  validated  by 
SMEs  and  compared  with  any  available  literature  and  supporting  documentation.  The  final  results 
were  also  peer-reviewed. 

3.6  Interviews 

As  the  AER  was  focused  on  strategic  issues  related  to  V2010  security  planning  and  operations,  the 
interviewees  were  selected  from  the  population  of  senior  officials  representing  all  of  the  main  partner 
organizations  and  who  operated  at  the  strategic-operational  interface.  The  interviewees  held  a  unique 
perspective,  which  enabled  them  to  provide  views  on  the  impact  of  strategic  decisions  on  operations, 
as  well  as  the  influence  of  operations  on  strategy.  Initially,  30  potential  interviewees  were  selected 
based  on  their  knowledge  of  their  respective  departments  and  agencies,  as  well  as  their  personal 
involvement  in  planning  and  operations.  The  target  group  of  interviewees  was  eventually  expanded  to 
41,  when  it  was  determined  that  the  breadth  of  partner  departments  and  agencies,  as  well  as  the 
diversity  of  roles,  could  not  be  covered  by  a  smaller  sample. 

The  criteria  for  the  choice  of  analysts  who  conducted  the  interviews  and  performed  the  subsequent 
analysis  was  also  important.  The  interviewers  consisted  of  three  operational  research  scientists  and 
the  primary  author  of  this  paper  with  a  background  in  knowledge  management.  Although  all  the 
interviewers  came  from  Defence  R&D  Canada,  an  organization  which  was  heavily  involved  in  V2010 
security  planning  and  operations,  none  of  the  individuals  chosen  for  the  AER  work  were  involved  in 
those  activities.  As  such,  the  interviewers  had  little  prior  knowledge  of  the  topic  of  the  study,  thus 
ensuring  external,  independent  and  unbiased  analysis. 

The  interviews  were  guided  by  a  questionnaire  which  was  developed  through  analysis  of  critical 
issues  as  described  above.  The  interviewees  were  asked  to: 

Please  reflect  on  the  strategic  and  operational  experience  that  you  gained  from  your 

involvement  in  planning  and  operations  for  V2010  when  answering  the  following  questions. 

1.  From  a  whole-of-government  perspective,  what  did  you  learn  about  planning  and 
operations  for  major  events? 

2.  Do  you  have  any  particular  suggestions  or  observations  pertaining  to  federal  whole-of- 
government  major  event  planning  and  operations  in  the  following  areas: 

a)  Governance? 

b)  Inter-agency  relationships? 
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c)  Sensitive  information  sharing? 

cl)  Assessment  and  protection  of  national,  provincial  and  regional  critical  infrastructure? 


3.  How  do  existing  federal  government  mandates  enable  or  hinder  the  whole-of-government 
public  safety  and  security  approach  for  major  events? 

4.  Can  you  describe  an  example  of  learning  or  innovation  that  occurred  within  your 
organization  as  a  result  of  inter-agency  involvement  in  planning  and  operations  for  the 
Games? 

5.  How  will  your  organization  make  use  of  what  it  has  learned  for  future,  major  inter-agency 
events? 

Most  of  the  interviews  were  recorded  and  all  of  the  recordings  were  transcribed.  Both  the  recordings 
and  the  transcriptions  were  labeled  using  codes  to  protect  the  confidentiality  of  the  respondents. 
Once  transcribed,  the  recordings  were  destroyed. 

Initially,  the  interviews  were  quite  structured,  closely  following  the  questionnaire;  however,  over  time 
they  evolved  to  a  less  structured  format.  More  probing  questions  emerged  as  the  interviewer’s 
experience  and  level  of  comfort  with  the  topics  at  hand  grew.  Furthermore,  it  became  apparent  that 
flexibility  and  adaptability  was  needed,  based  on  the  particular  experiences  of  individual  interviewees 
as  well  as  their  personalities,  and  the  culture  and  operational  model  of  their  organizations.  For 
example,  interviewees  who  came  from  “uniformed”  police  and  military  organizations,  which  tend  to 
have  more  “tight”  and  formalized  cultures,  generally  preferred  the  structured  interview  format  and 
rarely  ventured  beyond  the  scope  of  the  questionnaire.  On  the  other  hand,  some  interviewees 
preferred  to  “tell  stories”  rather  than  strictly  follow  the  questionnaire. 

The  analysis  of  the  interview  data  consisted  of  coding  the  interview  transcripts  using  QSR  NVIVO 
software.  The  codes  were  developed  based  on  the  initially  defined  critical  issues  as  well  as  major 
themes  which  emerged  through  the  interview  process.  Seven  main  themes  were  identified  through 
extensive  discussion  and  collective  analysis  of  the  transcripts  by  the  analysts.  These  included  topics 
such  as  governance,  exercises  and  the  roles  of  culture  and  personality. 

To  ensure  that  all  analysts  consistently  interpreted  the  codes,  some  of  the  transcripts  were  analyzed 
by  more  than  one  individual,  and  then  compared  for  uniformity  of  coding.  Even  after  the  coding  work 
was  divided  up  among  the  analysts,  frequent  meetings  were  held  to  discuss  the  findings  and  progress 
and  to  ensure  consistency  of  the  analysis  approach.  Once  all  the  transcripts  were  coded,  the 
information  was  sorted  according  to  individual  codes  and  divided  up  among  the  analysts  responsible 
for  their  respective  issues. 

Throughout  the  interviews  and  the  subsequent  analysis,  care  was  taken  to  protect  the  privacy  of  the 
interviewees.  It  was  important  for  the  study  that  the  interviewees  spoke  on  the  condition  of  anonymity, 
with  the  intention  that  this  would  provide  more  candid  information  and  allow  individuals  to  address 
issues  which  they  may  otherwise  not  be  comfortable  discussing.  To  ensure  that  the  identity  of  the 
interviewees  was  protected,  all  electronic  files  were  labeled  with  randomized  codes  and  precautions 
were  taken  with  data  storage  and  sharing.  All  the  recordings,  transcriptions,  and  notes  taken  by  the 
interviewers  were  destroyed  after  they  were  analyzed  and  the  relevant  information  extracted. 
Furthermore,  the  interview  results  were  anonymized  to  the  point  where  they  could  not  be  attributed  to 
any  single  individual.  For  example,  any  direct  quotations  used  in  the  analysis  were  stripped  of  any 
detail  which  could  potentially  indentify  an  individual. 

3.7  Social  Network  Analysis 
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Social  network  analysis  (SNA)  is  the  study  of  relationships  between  people  and  organizations.  By 
compiling  individual  relationship  links  between  members  of  a  group,  a  network  is  derived  that  can  be 
analyzed  to  understand  interactions  and  positions  within  a  social  group.  An  SNA  was  undertaken  as 
part  of  the  AER  to  identify  information-sharing  relationships  that  existed  between  the  people  who  were 
involved  in  V2010  security  planning  and  operations.  Given  that  individuals  represented  different 
departments  and  agencies,  the  SNA  provided  a  means  of  assessing  the  state  of  whole-of-government 
cooperation. 

The  following  overarching  research  questions  were  used  to  guide  the  SNA  and  provide  a  structure  for 
the  analysis: 

•  Are  the  people  and  organizations  that  should  be  contact  with  one  another  actually  in 
contact  to  facilitate  major  events  security  planning? 

•  Are  the  interactions  the  organizations  reported  in  the  social  network  analysis  consistent 
with  the  interview  findings? 

•  Which  are  the  central  organizations  in  the  network? 

Forty  interviewees  completed  the  SNA  portion  of  the  study.  The  interviewees  were  asked  to  record 
as  many  names  as  desired  based  on  their  response  to  the  following  question:  “Who  were  the  key 
people  with  whom  you  shared  information  on  V2010  planning  and  operations?”  Since  the  question 
was  posed  during  the  interviews  it  was  designed  to  be  very  general  in  order  not  to  limit  free  recall. 
While  information  sharing  typically  implies  both  transmitting  and  receiving  information,  the  question 
did  not  specify  this  directly  nor  did  it  ask  the  respondents  to  specify  the  frequency  of  interaction.  The 
interpretation  was  left  up  to  the  individual  interviewees,  and  therefore  the  only  thing  that  could  be 
established  with  certainty  from  the  data  collected,  was  whether  an  information  sharing  relationship 
was  present  or  not.  The  40  respondents  nominated  a  total  of  341  key  contacts  that  were  assembled 
into  a  social  network  which  is  shown  in  Figure  2. 
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Figure  2:  Social  network  diagram  representing  key  person-to-person  information  sharing 
relationships  during  V2010  security  planning  and  operations. 


The  person-to-person  information  sharing  network  gave  insight  into  relationships  between  V2010 
security  and  safety  stakeholders  and  analysis  of  the  positioning  of  individuals  in  the  network  provided 
valuable  information  about  that  the  different  roles  the  individuals  played  in  the  network.  For  example, 
analysis  revealed  that  some  individuals  served  as  a  bridge,  or  information  sharing  conduit,  connecting 
otherwise  disconnected  individuals.  The  SNA  also  revealed  that  approximately  55%  of  the 
interviewees  appeared  as  central  figures  in  the  network,  confirming  the  appropriate  choice  of  the 
sample  population.  A  separate  analysis  was  undertaken  that  looked  at  key  information  sharing 
relationships  between  the  organizational  entities,  as  opposed  to  individuals.  This  analysis  revealed 
which  organizations  served  as  intermediaries,  linking  together  different  communities,  and  which 
operated  more  on  the  periphery  of  the  network. 

While  the  data  collected  through  the  SNA  proved  very  valuable  it  is  important  to  recognize  its 
limitations.  Firstly,  data  was  obtained  from  the  40  interviewees  only  therefore  the  scope  of  the 
analysis  was  necessarily  limited  to  a  specific  category  of  individual.  Secondly,  the  data  was  collected 
at  the  level  of  the  individual  and  therefore  there  was  a  possibility  that  some  interviewees  may  have 
overstated  their  connections  in  order  to  portray  themselves  in  a  positive  light.  To  counteract  this 
tendency,  during  the  interviews,  the  interviewers  sought  to  limit  the  appearance  of  expectations 
concerning  the  number  of  identified  contacts  and  emphasized  the  need  to  convey  only  key  contacts. 
Another  limitation  is  that  missing  or  incomplete  data  was  especially  problematic  as  it  could  lead  to 
skewed  results  with  some  individuals  being  either  underrepresented  or  overemphasized.  To  address 
this  issue,  a  follow-up  on-line  survey  was  administered  to  gather  social  network  data  from  72 
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additional  individuals  who  were  not  interviewed,  but  who  were  identified  by  more  than  one  of  the 
interviewees  as  being  key  contacts.  Notwithstanding  the  limitations  of  the  data,  the  SNA  portion  of  the 
AER  provided  valuable  insight  into  the  information  sharing  relationships  which  existed  within  the 
security  and  safety  communities  during  V2010  planning  and  operations.  The  results  both 
complemented  and  supported  the  interview  data. 

3.7  Case  Studies 

Three  case  studies  were  selected  as  topics  which  would  provide  a  more  in-depth  exploration  of  the 
critical  issues.  Both  CBRNE  preparedness  and  Critical  Infrastructure  Protection  were  challenging 
cross-cutting  topics  which  involved  multiple  organizations  and  inter-jurisdictional  discussions.  By 
exploring  them  in  more  detail,  it  was  anticipated  that  the  analysis  could  indicate  how  these  topics 
might  be  managed  in  future  event  planning  and  in  normal  operations.  The  Olympic  Marine 
Operations  Centre  was  selected  as  a  case  study  illustrating  inter-agency  cooperative  information 
sharing  because  it  was  one  of  the  operational  centres  which  were  considered  to  be  particularly 
successful  in  managing  organizational  information  sharing  barriers.  Although  more  in-depth,  the  case 
studies  followed  the  same  analytical  pattern  as  the  issues  analysis,  including  a  capability  analysis. 

Some  of  the  case  study  data  was  obtained  during  the  initial  interviews  with  officials  who  were  directly 
involved  in  the  three  chosen  topics,  however  many  follow-up  interviews  were  scheduled  with  other 
pertinent  individuals  who  were  not  a  part  of  the  original  group  of  41  interviewees.  One  issue  which 
arose,  because  of  the  large  number  of  individuals  from  whom  input  was  solicited,  was  that  at  times 
diverging  accounts  and  opinions  of  certain  outcomes  were  recalled.  This  put  added  pressure  on  the 
analysts  to  fact-check  and  validate  all  accounts.  Another  challenge  was  that  DRDC  staff  had  been 
heavily  involved  in  each  of  the  three  case  study  areas  in  the  planning  leading  up  to  V201 0  and  it  was 
recognized  that  there  might  be  a  danger  of  bias  in  the  analysis.  To  compensate  for  this,  the  analysts 
were  required  to  validate  their  findings  with  non-DRDC  SMEs. 

4.  Discussion 

While  the  conclusions  of  the  study  have  not  been  disseminated  and  cannot  be  shared  outside  of  the 
stakeholder  group,  the  results  have  shown  that  the  challenges  of  whole-of-government  initiatives  most 
often  involve  complex  or  systemic  issues  that  are  not  easily  resolved.  The  AER  itself,  however, 
illustrated  both  benefits  and  challenges  to  the  Lessons  Learned  process  as  it  pertains  to  meta- 
organizational  learning. 

Firstly,  the  validity  of  the  conclusions  of  a  Lessons  Learned  process  is  heavily  dependent  on  the 
reliability  of  the  analysis.  The  AER’s  concentration  on  the  analytical  approach  was  able  to  produce 
results  that  were  considered  rigorous  and  reliable  by  the  stakeholders.  The  analysis  relied  upon  the 
“evidence”  provided  by  the  subjects;  the  views  had  to  be  validated,  first  by  complimentary  evidence 
from  other  subjects  and  secondly  by  independent  SMEs.  Similarly,  potential  solutions  posited  by  the 
analytical  team  had  to  be  verified  and  confirmed  by  a  diversity  of  SMEs.  This  approach  provided  the 
stakeholders  with  confidence  that  the  analysis  was  reliable  and  not  subjective. 

Secondly,  the  capability  assessment  provided  a  mechanism  against  which  to  compare,  even 
benchmark,  whole-of-government  capability.  Without  the  use  of  such  tools  as  the  TCL,  it  would  have 
been  difficult  to  determine  how  future  initiatives  might  be  sustained  or  improved.  The  TCL  is  a 
comprehensive  list  of  best  practices  in  the  public  safety  and  security  domain  and  was  able  to  serve  as 
a  de  facto  standard  for  some  of  the  issues,  e.g.,  information  sharing,  planning  and  exercises. 
Unfortunately,  such  capability  lists  are  not  available  for  all  aspects  of  meta-organizational  issues  and 
literature  reviews  were  undertaken  to  provide  guidance  for  other  topics,  e.g.,  governance,  culture  and 
personality  (Lemyre,  2010). 
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The  most  significant  challenge  to  the  AER  as  a  meta-organizational  Lessons  Learned  process  was 
the  need  to  engage  all  of  the  partner  agencies  in  a  non-threatening  way.  The  participant 
organizations  needed  to  be  assured  that  they  were  not  be  evaluated  or  judged.  This  was 
accomplished  through  extensive  communications  in  explaining  the  objective  of  a  Lessons  Learned 
process  to  be  “no  fault”  and  to  search  for  issues  of  mutual  concern.  The  consultation  and 
engagement  of  SMEs  throughout  the  process  ensured  that  delicate  issues  were  handled  sensitively 
and  that  assumptions  were  appropriately  questioned. 

In  terms  of  the  challenges  associated  with  integrating  Lessons  Learned  processes  in  a  whole-of- 
government  context,  while  similar  to  those  of  individual  organizations,  they  are  exacerbated  by  the 
complexity  posed  in  meta-organizations.  For  lessons  to  be  truly  learned,  organizational  and 
behavioural  change  can  only  result  when  a  change  authority  accepts  the  proposed  best  practices  or 
corrective  actions  and  directs  that  they  are  implemented.  It  may  be  difficult  to  determine  who  the 
appropriate  change  authority  is  in  an  individual  organization  and  even  more  difficult  to  convince  that 
individual  of  the  need  for  implementation.  In  a  whole-of-government  lessons  learned  process,  it  is 
more  complicated  to  determine  who  should,  or  even  can,  influence  or  direct  change.  Solid  analysis, 
therefore,  is  critical  to  positioning  the  results  for  endorsement  by  the  change  authorities  in  situations 
where  they  could  impact  many  organizations. 

5.  Conclusion 

The  AER  process  described  here  covered  only  the  first  three  stages  of  the  Lessons  Learned  Cycle. 
Each  stage  of  the  Cycle  poses  its  own  challenges  and  the  analysis  phase  can  often  be  overlooked  or 
underestimated  in  traditional  Lessons  Learned  processes.  The  AER  applied  a  robust  approach  to  the 
analysis  stage  with  a  four  methodology  approach:  document  and  literature  review,  interviews,  social 
network  analysis  and  case  studies.  The  capability  analysis  provided  an  additional  benchmarking 
framework.  It  is  through  this  robust  analytical  process  that  stakeholders  can  be  assured  of  the 
thoroughness  of  the  results  and  can  therefore  make  informed  decisions  on  complex  issues. 
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